Ranking Text Units According to Textual Saliency, Connectivity and Topic Aptness

An efficient use of lexical cohesion is described for ranking text units according to their contribution in defining the meaning of a text (textual saliency), their ability to form a cohesive sub-text (textual connectivity) and the extent and effectiveness to which they address the different topics which characterize the subject matter of the text (topic aptness). A specific application is also discussed where the method described is employed to build the indexing component of a summarization system to provide both generic and query-based indicative summaries. 1 Introduction As information systems become a more integral part of personal computing, it appears clear that summarization technology must be able to address users' needs effectively if it is to meet the demands of a growing market in the area of document management. MinimaUy, the abridgement of a text according to a user's needs involves selecting the most salient por-tiolts of the text which are topically best suited to represent the user's interests. This selection must also take into consideration the degree of connectivity among the chosen text portions so as to minimize the danger of producing summaries which contain poorly linked sentences. In addition, the assessment of textual saliency, connectivity and topic aptness must be computed efficiently enough so that summa-dens, Ian Johnson and Victor Poznaliski for helpful comments on previous versions of this document. Many thanks also to Stephen Burns for internet programming support, tlalf St.einberger for assistance in dictionary conversion, and Charlotte Boynton for editorial help. rization can be conveniently performed on-line. The goal of this paper is to show how these objectives can be achieved through a conceptual indexing technique based on an efficient use of lexical cohesion. 2 Background Lexical cohesion has been widely used in text analysis for the cornparative assessment of saliency and connectivity of text fragments. Following Ho W (1991), a simple way of computing lexical cohesion ill a text is to segment tile text into units (e.g sentences) and to count non-stop words I which co-occur in each pair of distinct text units, as shown in Table 2 for the text in Table 1. Text units which contain a greater number of shared non-stop words are more likely to provide a better abridgement of the original text for two reasons: • the more often a word with high informa-tional content occurs in a text, the more topical and germane to the text the word is …